{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zMGtqd7scIqM"
   },
   "outputs": [],
   "source": [
    "# Copyright 2019 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_lHs_6vFci8_"
   },
   "source": [
    "# Overview\n",
    "\n",
    "Time-series forecasting problems are ubiquitous throughout the business world and can be posed as a supervised machine learning problem. \n",
    "\n",
    "A common approach to creating features and labels is to use a sliding window where the features are historical entries and the label(s) represent entries in the future. As any data-scientist that works with time-series knows, this sliding window approach can be tricky to get right. \n",
    "\n",
    "In this notebook we share a workflow to tackle time-series problems.\n",
    "\n",
    "## Dataset\n",
    "\n",
    "For this demo we will be using New York City real estate data obtained from [nyc.gov](https://www1.nyc.gov/site/finance/taxes/property-annualized-sales-update.page). The data starts in 2003. The data can be loaded into BigQuery with the following code:\n",
    "\n",
    "```python\n",
    "# Read data. Data was collected from nyc open data repository.\n",
    "import pandas as pd\n",
    "dfr = pd.read_csv('https://storage.googleapis.com/asl-testing/data/nyc_open_data_real_estate.csv')\n",
    "\n",
    "# Upload to BigQuery.\n",
    "PROJECT = 'YOUR-PROJECT-HERE'\n",
    "DATASET = 'nyc_real_estate'\n",
    "TABLE = 'residential_sales'\n",
    "dfr.to_gbq('{}.{}'.format(DATASET, TABLE), PROJECT)\n",
    "```\n",
    "\n",
    "## Objective\n",
    "\n",
    "The goal of the notebook is to show how to forecast using Pandas and BigQuery. The steps achieve in this notebook are the following:\n",
    "\n",
    "1 Building a machine learning (ML) forecasting model locally\n",
    "* Create features and labels on a subsample of data\n",
    "* Train a model using sklearn\n",
    "\n",
    "2 Building and scaling out a ML using Google BigQuery \n",
    "* Create features and labels on the full dataset using BigQuery.\n",
    "* Train the model on the entire dataset using BigQuery ML\n",
    "\n",
    "3 Building an advanced forecasting modeling using recurrent neural network (RNN) model\n",
    "* Create features and labels on the full dataset using BigQuery.\n",
    "* Train a model using TensorFlow\n",
    "\n",
    "\n",
    "## Costs \n",
    "\n",
    "This tutorial uses billable components of Google Cloud Platform (GCP):\n",
    "\n",
    "* BigQuery\n",
    "* Cloud storage\n",
    "* AI Platform\n",
    "\n",
    "The BigQuery and Cloud Storage costs are < \\$0.05 and the AI Platform training job uses approximately 0.68 ML units or ~\\$0.33.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vJI1gOsdZzXU"
   },
   "source": [
    "#  Pandas: Rolling window for time-series forecasting\n",
    "\n",
    "We have created a Pandas solution `create_rolling_features_label` function that automatically creates the features/label setup. This is suitable for smaller datasets and for local testing before training on the Cloud. And we have also created a BigQuery script that creates these rolling windows suitable for large datasets.\n",
    "\n",
    "![](https://github.com/GoogleCloudPlatform/training-data-analyst/blob/master/blogs/gcp_forecasting/rolling_window.gif?raw=true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ookHmxWEZzXV"
   },
   "source": [
    "## Data Exploration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IU8LiOC-xBvW"
   },
   "source": [
    "This notebook is self-contained so let's clone the training-data-analyst repo so we can have access to the feature and label creation functions in `time_series.py` and `scalable_time_series.py`. We'll be using the pandas_gbq package so make sure that it is installed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mLjvYD6P8EaX"
   },
   "outputs": [],
   "source": [
    "!pip3 install pandas-gbq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "ZwKWSX3ddqiK",
    "outputId": "04893e3d-17da-4040-8535-3559916a8e88"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Cloning into 'training-data-analyst'...\n",
      "Checking out files: 100% (3693/3693), done.\n"
     ]
    }
   ],
   "source": [
    "%%bash \n",
    "git clone https://github.com/GoogleCloudPlatform/training-data-analyst.git \\\n",
    " --depth 1\n",
    "\n",
    "cd training-data-analyst/blogs/gcp_forecasting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Hvp3tOqOZzXV"
   },
   "source": [
    "After cloning the above repo we can important pandas and our custom module `time_series.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hhEDDJ2rZzXW"
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import pandas as pd\n",
    "import pandas_gbq as gbq\n",
    "\n",
    "from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor\n",
    "from sklearn.linear_model import Ridge\n",
    "\n",
    "import time_series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "b-9OJGeAZzXY"
   },
   "outputs": [],
   "source": [
    "# Allow you to easily have Python variables in SQL query.\n",
    "from IPython.core.magic import register_cell_magic\n",
    "from IPython import get_ipython\n",
    "\n",
    "\n",
    "@register_cell_magic('with_globals')\n",
    "def with_globals(line, cell):\n",
    "    contents = cell.format(**globals())\n",
    "    if 'print' in line:\n",
    "        print(contents)\n",
    "    get_ipython().run_cell(contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HYMBJNkWZzXa"
   },
   "source": [
    "For this demo we will be using New York City real estate data obtained from [nyc.gov](https://www1.nyc.gov/site/finance/taxes/property-annualized-sales-update.page). This public dataset starts in 2003. The data can be loaded into BigQuery with the following code:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dKCs9YR_xnht"
   },
   "outputs": [],
   "source": [
    "dfr = pd.read_csv('https://storage.googleapis.com/asl-testing/data/nyc_open_data_real_estate.csv')\n",
    " \n",
    "# Upload to BigQuery.\n",
    "PROJECT = \"[your-project-id]\"\n",
    "DATASET = 'nyc_real_estate'\n",
    "TABLE = 'residential_sales'\n",
    "BUCKET = \"[your-bucket]\"  # Used later.\n",
    "gbq.to_gbq(dfr, '{}.{}'.format(DATASET, TABLE), PROJECT, if_exists='replace')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ipj8LLx8xzra"
   },
   "source": [
    "Since we are just doing local modeling, let's just use a subsample of the data. Later we will train on all of the data in the cloud."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bYyydgxZx-4l"
   },
   "outputs": [],
   "source": [
    "SOURCE_TABLE = TABLE\n",
    "FILTER = '''residential_units = 1 AND sale_price > 10000 \n",
    "AND sale_date > TIMESTAMP('2010-12-31 00:00:00')'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GkCh82kBZzXd"
   },
   "outputs": [],
   "source": [
    "%%with_globals\n",
    "%%bigquery --project {PROJECT} df\n",
    "SELECT \n",
    "  borough,\n",
    "  neighborhood,\n",
    "  building_class_category,\n",
    "  tax_class_at_present,\n",
    "  block,\n",
    "  lot,\n",
    "  ease_ment,\n",
    "  building_class_at_present,\n",
    "  address,\n",
    "  apartment_number,\n",
    "  zip_code,\n",
    "  residential_units,\n",
    "  commercial_units,total_units,\n",
    "  land_square_feet,\n",
    "  gross_square_feet,\n",
    "  year_built,\n",
    "  tax_class_at_time_of_sale,\n",
    "  building_class_at_time_of_sale,\n",
    "  sale_price,\n",
    "  sale_date,\n",
    "  price_per_sq_ft\n",
    "FROM\n",
    "  {SOURCE_TABLE}\n",
    "WHERE\n",
    "  {FILTER}\n",
    "ORDER BY\n",
    "  sale_date\n",
    "LIMIT\n",
    "  100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SK_f0WbyZzXf"
   },
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dIumN3ICZzXi"
   },
   "outputs": [],
   "source": [
    "%%with_globals\n",
    "%%bigquery --project {PROJECT} df\n",
    "SELECT\n",
    "  neighborhood,\n",
    "  COUNT(*) AS cnt\n",
    "FROM\n",
    "  {SOURCE_TABLE}\n",
    "WHERE\n",
    "  {FILTER}\n",
    "GROUP BY\n",
    "  neighborhood\n",
    "ORDER BY\n",
    "  cnt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vRQ_SUK3ZzXk"
   },
   "source": [
    "The most sales are from the upper west side, midtown west, and the upper east side."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SmA_5KEeZzXl"
   },
   "outputs": [],
   "source": [
    "ax = df.set_index('neighborhood').cnt\\\n",
    "    .tail(10)\\\n",
    "    .plot(kind='barh');\n",
    "ax.set_xlabel('total sales');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ClThdNOFZzXn"
   },
   "source": [
    "SOHO and Civic Center are the most expensive neighborhoods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "a1QIaZ0zZzXo"
   },
   "outputs": [],
   "source": [
    "%%with_globals\n",
    "%%bigquery --project {PROJECT} df\n",
    "SELECT\n",
    "  neighborhood,\n",
    "  APPROX_QUANTILES(sale_price, 100)[\n",
    "OFFSET\n",
    "  (50)] AS median_price\n",
    "FROM\n",
    "  {SOURCE_TABLE}\n",
    "WHERE\n",
    "  {FILTER}\n",
    "GROUP BY\n",
    "  neighborhood\n",
    "ORDER BY\n",
    "  median_price"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ei_ClN6KZzXr"
   },
   "outputs": [],
   "source": [
    "ax = df.set_index('neighborhood').median_price\\\n",
    "    .tail(10)\\\n",
    "    .plot(kind='barh');\n",
    "ax.set_xlabel('median price')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VtqoGYDqZzXt"
   },
   "source": [
    "## Build features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YnxFfGHQZzXu"
   },
   "source": [
    "Let's create features for building a machine learning model:\n",
    "\n",
    "1. Aggregate median sales for each week. Prices are noisy and by grouping by week, we will smooth out irregularities.\n",
    "2. Create a rolling window to split the single long time series into smaller windows. One feature vector will contain a single window and the label will be a single observation (or window for multiple predictions) occuring after the window."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RsPy6L-LZzXv"
   },
   "outputs": [],
   "source": [
    "%%with_globals\n",
    "%%bigquery --project asl-testing-217717 df\n",
    "SELECT\n",
    "  sale_week,\n",
    "  APPROX_QUANTILES(sale_price, 100)[\n",
    "OFFSET\n",
    "  (50)] AS median_price\n",
    "FROM (\n",
    "  SELECT\n",
    "    TIMESTAMP_TRUNC(sale_date, week) AS sale_week,\n",
    "    sale_price\n",
    "  FROM\n",
    "    {SOURCE_TABLE}\n",
    "  WHERE\n",
    "    {FILTER})\n",
    "GROUP BY\n",
    "  sale_week\n",
    "ORDER BY\n",
    "  sale_week"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MSc6uxdpZzXw"
   },
   "outputs": [],
   "source": [
    "sales = pd.Series(df.median_price)\n",
    "sales.index= pd.DatetimeIndex(df.sale_week.dt.date)\n",
    "sales.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GYCDApw1ZzXy"
   },
   "outputs": [],
   "source": [
    "ax = sales.plot(figsize=(8,4), label='median_price')\n",
    "ax = sales.rolling(10).mean().plot(ax=ax, label='10 week rolling average')\n",
    "ax.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kHzopDYiZzX1"
   },
   "source": [
    "### Sliding window"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PfKIY0uzZzX1"
   },
   "source": [
    "Let's create our features. We will use the `create_rolling_features_label` function that automatically creates the features/label setup.\n",
    "\n",
    "![](rolling_window.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "M9iOPEK6ZzX2"
   },
   "source": [
    "Create the features and labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XptKazGzZzX2"
   },
   "outputs": [],
   "source": [
    "WINDOW_SIZE = 52 * 1\n",
    "HORIZON = 4*6\n",
    "MONTHS = 0\n",
    "WEEKS = 1\n",
    "LABELS_SIZE = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Yh0tnWBZzfOK"
   },
   "outputs": [],
   "source": [
    "df = time_series.create_rolling_features_label(sales, window_size=WINDOW_SIZE, pred_offset=HORIZON)\n",
    "df = time_series.add_date_features(df, df.index, months=MONTHS, weeks=WEEKS)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Tu4M1IwMZzX5"
   },
   "source": [
    "Let's train our model using all weekly median prices from 2003 -- 2015. Then we will test our model's performance on prices from 2016 -- 2018"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lv5SmKavZzX6"
   },
   "outputs": [],
   "source": [
    "# Features, label.\n",
    "X = df.drop('label', axis=1)\n",
    "y = df['label']\n",
    "\n",
    "# Train/test split. Splitting on time.\n",
    "train_ix = time_series.is_between_dates(y.index,\n",
    "                                  end='2015-12-30')\n",
    "test_ix = time_series.is_between_dates(y.index,\n",
    "                                 start='2015-12-30',\n",
    "                                 end='2018-12-30 08:00:00')\n",
    "X_train, y_train = X.iloc[train_ix], y.iloc[train_ix]\n",
    "X_test, y_test = X.iloc[test_ix], y.iloc[test_ix]\n",
    "\n",
    "print(X_train.shape, X_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QBqi5AeQZzX8"
   },
   "source": [
    "Z-score normalization for the features for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IZbiHq17ZzX9"
   },
   "outputs": [],
   "source": [
    "mean = X_train.mean()\n",
    "std = X_train.std()\n",
    "\n",
    "def zscore(X):\n",
    "    return (X-mean)/std\n",
    "\n",
    "X_train = zscore(X_train)\n",
    "X_test = zscore(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hOAemgQjZzX_"
   },
   "source": [
    "## Initial model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oehN8bukZzYA"
   },
   "source": [
    "### Baseline\n",
    "Build naive model that just uses the mean of training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nQelQOjHZzYB"
   },
   "outputs": [],
   "source": [
    "df_baseline = y_test.to_frame(name='label')\n",
    "df_baseline['pred'] = y_train.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YNfta4KeZzYE"
   },
   "outputs": [],
   "source": [
    "# Join mean predictions with test labels.\n",
    "baseline_global_metrics = time_series.Metrics(df_baseline.pred,\n",
    "                                              df_baseline.label)\n",
    "baseline_global_metrics.report(\"Global Baseline Model\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ghq-_e-dZzYM"
   },
   "outputs": [],
   "source": [
    "# Train model.\n",
    "cl = RandomForestRegressor(n_estimators=500, max_features='sqrt',\n",
    "                              random_state=10, criterion='mse')\n",
    "cl = Ridge(100)\n",
    "cl = GradientBoostingRegressor ()\n",
    "\n",
    "cl.fit(X_train, y_train)\n",
    "pred = cl.predict(X_test)\n",
    "random_forest_metrics = time_series.Metrics(y_test,\n",
    "                                            pred)\n",
    "random_forest_metrics.report(\"Forest Model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fMVazQAXZzYQ"
   },
   "source": [
    "The regression model performs 35% better than the baseline model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NfzwRXbgZzYQ"
   },
   "source": [
    "Observations:\n",
    "* Linear Regression does okay for this dataset (Regularization helps generalize the model)\n",
    "* Random Forest is better -- doesn't require a lot of tuning. It performs a bit better than regression.\n",
    "* Gradient Boosting does do better than regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QBWwnYocZzYR"
   },
   "source": [
    "## Interpret results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JzWG0Br8ZzYS"
   },
   "outputs": [],
   "source": [
    "# Data frame to query for plotting\n",
    "df_res = pd.DataFrame({'pred': pred, 'baseline': df_baseline.pred, 'y_test': y_test})\n",
    "metrics = time_series.Metrics(df_res.y_test, df_res.pred)\n",
    "ax = df_res.iloc[:].plot(y=[ 'pred', 'y_test'],\n",
    "                 style=['b-','k-'],\n",
    "                 figsize=(10,5))\n",
    "ax.set_title('rmse: {:2.2f}'.format(metrics.rmse), size=16);\n",
    "ax.set_ylim(20,)\n",
    "df_res.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Tc0iBrvMZzYV"
   },
   "source": [
    "# BigQuery modeling\n",
    "\n",
    "We have observed there is signal in our data and our smaller, local model is working better. Let's scale this model out to the cloud. Let's train a BigQuery Machine Learning (BQML) on the full dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "C8NFh5XB1_wI"
   },
   "source": [
    "## Set up your GCP project\n",
    "\n",
    "**The following steps are required, regardless of your notebook environment.**\n",
    "\n",
    "1. [Select or create a GCP project.](https://console.cloud.google.com/cloud-resource-manager)\n",
    "\n",
    "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
    "\n",
    "3. BigQuery is automatically enabled in new projects. [To activate BigQuery in a pre-existing project, go to Enable the BigQuery API.](https://console.cloud.google.com/flows/enableapi?apiid=bigquery)\n",
    "\n",
    "4. Enter your project ID in the cell below.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yB8uSgAQ2AHL"
   },
   "source": [
    "### Authenticate your GCP account\n",
    "\n",
    "**If you are using AI Platform Notebooks**, your environment is already\n",
    "authenticated. Skip this step.\n",
    "\n",
    "**If you are using Colab**, run the cell below and follow the instructions\n",
    "when prompted to authenticate your account via oAuth.\n",
    "\n",
    "**Otherwise**, follow these steps:\n",
    "\n",
    "1. In the GCP Console, go to the [**Create service account key**\n",
    "   page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
    "\n",
    "2. From the **Service account** drop-down list, select **New service account**.\n",
    "\n",
    "3. In the **Service account name** field, enter a name.\n",
    "\n",
    "4. From the **Role** drop-down list, select\n",
    "   **BigQuery > BigQuery Admin** and\n",
    "   **Storage > Storage Object Admin**.\n",
    "\n",
    "5. Click *Create*. A JSON file that contains your key downloads to your\n",
    "computer.\n",
    "\n",
    "6. Enter the path to your service account key as the `GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-kOrbyi82FxO"
   },
   "source": [
    "### Import libraries\n",
    "Import supporting modules:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "g_NXUrEyZzYW"
   },
   "outputs": [],
   "source": [
    "# Import BigQuery module\n",
    "from google.cloud import bigquery\n",
    "\n",
    "# Import external custom module containing SQL queries\n",
    "import scalable_time_series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "H7rOvVhuZzYY"
   },
   "outputs": [],
   "source": [
    "# Define hyperparameters\n",
    "value_name = \"med_sales_price\"\n",
    "downsample_size = 7 # 7 days into 1 week\n",
    "window_size = 52\n",
    "labels_size = 1\n",
    "horizon = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Fb-WZHnSZzYZ"
   },
   "outputs": [],
   "source": [
    "# Construct a BigQuery client object.\n",
    "client = bigquery.Client()\n",
    "\n",
    "# Set dataset_id to the ID of the dataset to create.\n",
    "sink_dataset_name = \"temp_forecasting_dataset\"\n",
    "dataset_id = \"{}.{}\".format(client.project, sink_dataset_name)\n",
    "\n",
    "# Construct a full Dataset object to send to the API.\n",
    "dataset = bigquery.Dataset.from_string(dataset_id)\n",
    "\n",
    "# Specify the geographic location where the dataset should reside.\n",
    "dataset.location = \"US\"\n",
    "\n",
    "# Send the dataset to the API for creation.\n",
    "# Raises google.api_core.exceptions.Conflict if the Dataset already\n",
    "# exists within the project.\n",
    "try:\n",
    "    dataset = client.create_dataset(dataset)  # API request\n",
    "    print(\"Created dataset {}.{}\".format(client.project, dataset.dataset_id))\n",
    "except Exception as e:\n",
    "    print(\"Dataset {}.{} already exists\".format(\n",
    "        client.project, dataset.dataset_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gxWP2nveZzYb"
   },
   "source": [
    "We need to create a date range table in BigQuery so that we can join our data to that to get the correct sequences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "c83vELIxZzYb"
   },
   "outputs": [],
   "source": [
    "# Call BigQuery and examine in dataframe\n",
    "source_dataset = \"nyc_real_estate\"\n",
    "source_table_name = \"all_sales\"\n",
    "query_create_date_range = scalable_time_series.create_date_range(\n",
    "    client.project, source_dataset, source_table_name)\n",
    "df = client.query(query_create_date_range + \"LIMIT 100\").to_dataframe()\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9neEv_RGZzYd"
   },
   "source": [
    "Execute query and write to BigQuery table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dJ8t3DaFZzYd"
   },
   "outputs": [],
   "source": [
    "job_config = bigquery.QueryJobConfig()\n",
    "# Set the destination table\n",
    "table_name = \"start_end_timescale_date_range\"\n",
    "table_ref = client.dataset(sink_dataset_name).table(table_name)\n",
    "job_config.destination = table_ref\n",
    "job_config.write_disposition = \"WRITE_TRUNCATE\"\n",
    "\n",
    "# Start the query, passing in the extra configuration.\n",
    "query_job = client.query(\n",
    "    query=query_create_date_range,\n",
    "    # Location must match that of the dataset(s) referenced in the query\n",
    "    # and of the destination table.\n",
    "    location=\"US\",\n",
    "    job_config=job_config)  # API request - starts the query\n",
    "\n",
    "query_job.result()  # Waits for the query to finish\n",
    "print(\"Query results loaded to table {}\".format(table_ref.path))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AU_--fnmZzYg"
   },
   "source": [
    "Now that we have the date range table created we can create our training dataset for BQML."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wDvoBA4KZzYg"
   },
   "outputs": [],
   "source": [
    "# Call BigQuery and examine in dataframe\n",
    "sales_dataset_table = source_dataset + \".\" + source_table_name\n",
    "query_bq_sub_sequences = scalable_time_series.bq_create_rolling_features_label(\n",
    "    client.project, sink_dataset_name, table_name, sales_dataset_table,\n",
    "    value_name, downsample_size, window_size, horizon, labels_size)\n",
    "print(query_bq_sub_sequences[0:500])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5zCew2cLZzYi"
   },
   "outputs": [],
   "source": [
    "%%with_globals\n",
    "%%bigquery --project $PROJECT\n",
    "{query_bq_sub_sequences}\n",
    "LIMIT 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8q2cOLr_ZzYk"
   },
   "source": [
    "## Create BigQuery dataset\n",
    "\n",
    "Prior to now we've just been reading an existing BigQuery table, now we're going to create our own so so we need some place to put it. In BigQuery parlance, `Dataset` means a folder for tables. \n",
    "\n",
    "We will take advantage of BigQuery's [Python Client](https://cloud.google.com/bigquery/docs/reference/libraries#client-libraries-install-python) to create the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7GEKOl8XZzYl"
   },
   "outputs": [],
   "source": [
    "bq = bigquery.Client(project = PROJECT)\n",
    "\n",
    "dataset = bigquery.Dataset(bq.dataset(\"bqml_forecasting\"))\n",
    "try:\n",
    "    bq.create_dataset(dataset) # will fail if dataset already exists\n",
    "    print(\"Dataset created\")\n",
    "except:\n",
    "    print(\"Dataset already exists\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zySA6iArZzYm"
   },
   "source": [
    "Split dataset into a train and eval set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Sklp2y9FZzYn"
   },
   "outputs": [],
   "source": [
    "feature_list = [\"price_ago_{time}\".format(time=time)\n",
    "                for time in range(window_size, 0, -1)]\n",
    "label_list = [\"price_ahead_{time}\".format(time=time)\n",
    "              for time in range(1, labels_size + 1)]\n",
    "select_list = \",\".join(feature_list + label_list)\n",
    "select_string = \"SELECT {select_list} FROM ({query})\".format(\n",
    "    select_list=select_list,\n",
    "    query=query_bq_sub_sequences)\n",
    "concat_vars = []\n",
    "concat_vars.append(\"CAST(feat_seq_start_date AS STRING)\")\n",
    "concat_vars.append(\"CAST(lab_seq_end_date AS STRING)\")\n",
    "farm_finger = \"FARM_FINGERPRINT(CONCAT({concat_vars}))\".format(\n",
    "    concat_vars=\", \".join(concat_vars))\n",
    "sampling_clause = \"ABS(MOD({farm_finger}, 100))\".format(\n",
    "    farm_finger=farm_finger)\n",
    "bqml_train_query = \"{select_string} WHERE {sampling_clause} < 80\".format(\n",
    "    select_string=select_string, sampling_clause=sampling_clause)\n",
    "bqml_eval_query = \"{select_string} WHERE {sampling_clause} >= 80\".format(\n",
    "    select_string=select_string, sampling_clause=sampling_clause)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VjmEEruYZzYp"
   },
   "source": [
    "## Create model\n",
    "\n",
    "To create a model\n",
    "1. Use `CREATE MODEL` and provide a destination table for resulting model. Alternatively we can use `CREATE OR REPLACE MODEL` which allows overwriting an existing model.\n",
    "2. Use `OPTIONS` to specify the model type (linear_reg or logistic_reg). There are many more options [we could specify](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-create#model_option_list), such as regularization and learning rate, but we'll accept the defaults.\n",
    "3. Provide the query which fetches the training data \n",
    "\n",
    "Have a look at [Step Two of this tutorial](https://cloud.google.com/bigquery/docs/bigqueryml-natality) to see another example.\n",
    "\n",
    "**The query will take about two minutes to complete**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CfcFZv6vZzYr"
   },
   "outputs": [],
   "source": [
    "%%with_globals\n",
    "%%bigquery --project $PROJECT\n",
    "CREATE or REPLACE MODEL bqml_forecasting.nyc_real_estate\n",
    "OPTIONS(model_type = \"linear_reg\",\n",
    "        input_label_cols = [\"price_ahead_1\"]) AS\n",
    "{bqml_train_query}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sveGIhdSZzYw"
   },
   "source": [
    "## Get training statistics\n",
    "\n",
    "Because the query uses a `CREATE MODEL` statement to create a table, you do not see query results. The output is an empty string.\n",
    "\n",
    "To get the training results we use the [`ML.TRAINING_INFO`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-train) function.\n",
    "\n",
    "Have a look at [Step Three and Four of this tutorial](https://cloud.google.com/bigquery/docs/bigqueryml-natality) to see a similar example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bU32XhyKZzYw"
   },
   "outputs": [],
   "source": [
    "%%bigquery --project $PROJECT\n",
    "SELECT\n",
    "    {select_list}\n",
    "FROM\n",
    "    ML.TRAINING_INFO(MODEL `bqml_forecasting.nyc_real_estate`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CHjeYKihZzYy"
   },
   "source": [
    "'eval_loss' is reported as mean squared error, so our RMSE is **291178**. Your results may vary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CtGWXRp0ZzYy"
   },
   "outputs": [],
   "source": [
    "%%with_globals\n",
    "%%bigquery --project $PROJECT\n",
    "#standardSQL\n",
    "SELECT\n",
    "  {select_list}\n",
    "FROM\n",
    "  ML.EVALUATE(MODEL `bqml_forecasting.nyc_real_estate`, ({bqml_eval_query}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VYmc-k-aZzY0"
   },
   "source": [
    "## Predict\n",
    "\n",
    "To use our model to make predictions, we use `ML.PREDICT`. Let's, use the `nyc_real_estate` you trained above to infer median sales price of all of our data.\n",
    "\n",
    "Have a look at [Step Five of this tutorial](https://cloud.google.com/bigquery/docs/bigqueryml-natality) to see another example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vE_xFGwqZzY0"
   },
   "outputs": [],
   "source": [
    "%%with_globals\n",
    "%%bigquery --project $PROJECT df\n",
    "#standardSQL\n",
    "SELECT\n",
    "    predicted_price_ahead_1\n",
    "FROM\n",
    "    ML.PREDICT(MODEL `bqml_forecasting.nyc_real_estate`, ({bqml_eval_query}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wLYqqxcIZzY2"
   },
   "source": [
    "# TensorFlow Sequence Model\n",
    "\n",
    "If you might want to use a more custom model, then Keras or TensorFlow may be helpful. Below we are going to create a custom LSTM sequence-to-one model that will read our input data in via CSV files and will train and evaluate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "A63Hr3rYZzY3"
   },
   "source": [
    "Create temporary BigQuery dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3FMjVCANZzY4"
   },
   "outputs": [],
   "source": [
    "# Construct a BigQuery client object.\n",
    "client = bigquery.Client()\n",
    "\n",
    "# Set dataset_id to the ID of the dataset to create.\n",
    "sink_dataset_name = \"temp_forecasting_dataset\"\n",
    "dataset_id = \"{}.{}\".format(client.project, sink_dataset_name)\n",
    "\n",
    "# Construct a full Dataset object to send to the API.\n",
    "dataset = bigquery.Dataset.from_string(dataset_id)\n",
    "\n",
    "# Specify the geographic location where the dataset should reside.\n",
    "dataset.location = \"US\"\n",
    "\n",
    "# Send the dataset to the API for creation.\n",
    "# Raises google.api_core.exceptions.Conflict if the Dataset already\n",
    "# exists within the project.\n",
    "try:\n",
    "    dataset = client.create_dataset(dataset)  # API request\n",
    "    print(\"Created dataset {}.{}\".format(client.project, dataset.dataset_id))\n",
    "except:\n",
    "    print(\"Dataset {}.{} already exists\".format(\n",
    "        client.project, dataset.dataset_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rR_fb27mZzY8"
   },
   "source": [
    "We need to create a date range table in BigQuery so that we can join our data to that to get the correct sequences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "j5SHGz8vZzY9"
   },
   "outputs": [],
   "source": [
    "# Call BigQuery and examine in dataframe\n",
    "source_dataset = \"nyc_real_estate\"\n",
    "source_table_name = \"all_sales\"\n",
    "query_create_date_range = scalable_time_series.create_date_range(\n",
    "    client.project, source_dataset, source_table_name)\n",
    "df = client.query(query_create_date_range + \"LIMIT 100\").to_dataframe()\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UPJQz2kJZzY-"
   },
   "source": [
    "Execute query and write to BigQuery table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "It2eeNHGZzZA"
   },
   "outputs": [],
   "source": [
    "job_config = bigquery.QueryJobConfig()\n",
    "# Set the destination table\n",
    "table_name = \"start_end_timescale_date_range\"\n",
    "table_ref = client.dataset(sink_dataset_name).table(table_name)\n",
    "job_config.destination = table_ref\n",
    "job_config.write_disposition = \"WRITE_TRUNCATE\"\n",
    "\n",
    "# Start the query, passing in the extra configuration.\n",
    "query_job = client.query(\n",
    "    query=query_create_date_range,\n",
    "    # Location must match that of the dataset(s) referenced in the query\n",
    "    # and of the destination table.\n",
    "    location=\"US\",\n",
    "    job_config=job_config)  # API request - starts the query\n",
    "\n",
    "query_job.result()  # Waits for the query to finish\n",
    "print(\"Query results loaded to table {}\".format(table_ref.path))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YqfnVenEZzZB"
   },
   "source": [
    "Now that we have the date range table created we can create our training dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0tb4n4UcZzZC"
   },
   "outputs": [],
   "source": [
    "# Call BigQuery and examine in dataframe\n",
    "sales_dataset_table = source_dataset + \".\" + source_table_name\n",
    "downsample_size = 7\n",
    "query_csv_sub_seqs = scalable_time_series.csv_create_rolling_features_label(\n",
    "    client.project, sink_dataset_name, table_name, sales_dataset_table,\n",
    "    value_name, downsample_size, window_size, horizon, labels_size)\n",
    "df = client.query(query_csv_sub_seqs + \"LIMIT 100\").to_dataframe()\n",
    "df.head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "P3WpXyrnZzZD"
   },
   "source": [
    "Now let's write the our training data table to BigQuery for train and eval so that we can export to CSV for TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JQH7I2rrZzZE"
   },
   "outputs": [],
   "source": [
    "job_config = bigquery.QueryJobConfig()\n",
    "csv_select_list = \"med_sales_price_agg, labels_agg\"\n",
    "for step in [\"train\", \"eval\"]:\n",
    "    if step == \"train\":\n",
    "        selquery = \"SELECT {csv_select_list} FROM ({}) WHERE {} < 80\".format(\n",
    "            query_csv_sub_sequences, sampling_clause)\n",
    "    else:\n",
    "        selquery = \"SELECT {csv_select_list} FROM ({}) WHERE {} >= 80\".format(\n",
    "            query_csv_sub_sequences, sampling_clause)\n",
    "    # Set the destination table\n",
    "    table_name = \"nyc_real_estate_{}\".format(step)\n",
    "    table_ref = client.dataset(sink_dataset_name).table(table_name)\n",
    "    job_config.destination = table_ref\n",
    "    job_config.write_disposition = \"WRITE_TRUNCATE\"\n",
    "\n",
    "    # Start the query, passing in the extra configuration.\n",
    "    query_job = client.query(\n",
    "        query=selquery,\n",
    "        # Location must match that of the dataset(s) referenced in the query\n",
    "        # and of the destination table.\n",
    "        location=\"US\",\n",
    "        job_config=job_config)  # API request - starts the query\n",
    "\n",
    "    query_job.result()  # Waits for the query to finish\n",
    "    print(\"Query results loaded to table {}\".format(table_ref.path))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CYzWMAvPZzZH"
   },
   "source": [
    "Export BigQuery table to CSV in GCS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "fnMxjziAZzZI"
   },
   "outputs": [],
   "source": [
    "dataset_ref = client.dataset(\n",
    "    dataset_id=sink_dataset_name, project=client.project)\n",
    "\n",
    "for step in [\"train\", \"eval\"]:\n",
    "    destination_uri = \"gs://{}/{}\".format(\n",
    "        BUCKET, \"forecasting/nyc_real_estate/data/{}*.csv\".format(step))\n",
    "    table_name = \"nyc_real_estate_{}\".format(step)\n",
    "    table_ref = dataset_ref.table(table_name)\n",
    "    extract_job = client.extract_table(\n",
    "        table_ref,\n",
    "        destination_uri,\n",
    "        # Location must match that of the source table.\n",
    "        location=\"US\",\n",
    "    )  # API request\n",
    "    extract_job.result()  # Waits for job to complete.\n",
    "\n",
    "    print(\"Exported {}:{}.{} to {}\".format(\n",
    "        client.project, sink_dataset_name, table_name, destination_uri))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xfcuJSfaZzZL"
   },
   "outputs": [],
   "source": [
"!gcloud storage cp gs://asl-testing-bucket/forecasting/nyc_real_estate/data/*.csv ."   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "14lhOTqGZzZM"
   },
   "outputs": [],
   "source": [
    "!head train*.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "o70diLtZZzZO"
   },
   "source": [
    "### Train TensorFlow on Google Cloud AI Platform."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cNthzw6sZzZO"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "PROJECT = PROJECT # REPLACE WITH YOUR PROJECT ID\n",
    "BUCKET = BUCKET # REPLACE WITH A BUCKET NAME\n",
    "REGION = \"us-central1\" # REPLACE WITH YOUR REGION e.g. us-central1\n",
    "\n",
    "# Import os environment variables\n",
    "os.environ[\"PROJECT\"] = PROJECT\n",
    "os.environ[\"BUCKET\"] =  BUCKET\n",
    "os.environ[\"REGION\"] = REGION\n",
    "os.environ[\"TF_VERSION\"] = \"1.13\"\n",
    "os.environ[\"SEQ_LEN\"] = str(WINDOW_SIZE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zAjcp0PXZzZQ"
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "OUTDIR=gs://$BUCKET/forecasting/nyc_real_estate/trained_model\n",
    "JOBNAME=nyc_real_estate$(date -u +%y%m%d_%H%M%S)\n",
    "echo $OUTDIR $REGION $JOBNAME\n",
    "gcloud storage rm --recursive --continue-on-error $OUTDIR\n",    "gcloud ai-platform jobs submit training $JOBNAME \\\n",
    "  --region=$REGION \\\n",
    "  --module-name=trainer.task \\\n",
    "  --package-path=$PWD/tf_module/trainer \\\n",
    "  --job-dir=$OUTDIR \\\n",
    "  --staging-bucket=gs://$BUCKET \\\n",
    "  --scale-tier=STANDARD_1 \\\n",
    "  --runtime-version=$TF_VERSION \\\n",
    "  -- \\\n",
    "  --train_file_pattern=\"gs://asl-testing-bucket/forecasting/nyc_real_estate/data/train*.csv\" \\\n",
    "  --eval_file_pattern=\"gs://asl-testing-bucket/forecasting/nyc_real_estate/data/eval*.csv\" \\\n",
    "  --output_dir=$OUTDIR \\\n",
    "  --job-dir=./tmp \\\n",
    "  --seq_len=$SEQ_LEN \\\n",
    "  --train_batch_size=32 \\\n",
    "  --eval_batch_size=32 \\\n",
    "  --train_steps=1000 \\\n",
    "  --learning_rate=0.01 \\\n",
    "  --start_delay_secs=60 \\\n",
    "  --throttle_secs=60 \\\n",
    "  --lstm_hidden_units=\"32,16,8\""
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "v2_gcp_time_series_forecasting.ipynb",
   "provenance": [],
   "toc_visible": true,
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
